Assignment: Project - 4: Unsupervised Learning¶
submitted by Subodh Nath Pushpak¶
The data contains features extracted from the silhouette of vehicles in different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.
Domain: Object recognition¶
Context¶
The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
Import the necessary libraries¶
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import math
import seaborn as sns
%matplotlib inline
from sklearn import metrics
from scipy.stats import zscore
from sklearn import model_selection
from sklearn.metrics import accuracy_score,f1_score,recall_score,precision_score, confusion_matrix
Task 1: Data pre-processing – Perform all necessary preprocessing on the data ready to be fed to an Unsupervised algorithm¶
Task 1 a. Load the Dataset and find the datatypes¶
In [2]:
# loading the data
dataFilePath = './vehicle.csv'
data_df= pd.read_csv(dataFilePath)
data_df.head()
Out[2]:
In [3]:
# shape of the data
rows, cols = data_df.shape
print (f"There are {rows} observations / rows and {cols} attributes / columns")
In [4]:
# Data type of each attribute
data_df.info() # Info on each of the attribute
In [5]:
data_df.columns
Out[5]:
Inference¶
- All are numeric in nature
- Out of total 19 columns, 14 are of float64 type, 4 are of int64 type and 1 is of object type
- 'class' column is our target column
- There are missing values in many columns like circularity, distance circularity, radius_ratio, pr.axis_aspect_ratio, etc.
Task 1 b. Missing Value identification and appropriate actions¶
In [6]:
# determine if any value is null
data_df.isnull().values.any()
Out[6]:
In [7]:
# determine the sum of na values
data_df.apply(lambda x : sum(x.isnull()))
Out[7]:
In [8]:
#List all the rows having missing value in any of the single or multiple columns
missing_values_cols=data_df.columns[data_df.isnull().any()]
data_df[data_df.isnull().any(axis=1)][missing_values_cols].head()
Out[8]:
In [9]:
missingRows, missingCols = data_df[data_df.isnull().any(axis=1)][missing_values_cols].shape
print (f"There are {missingRows} rows with {missingCols} missing columns")
Inference¶
- There are null values in some of the features
- There are 33 missing rows and 14 missing columns. 'radius_ratio' and 'skewness_about' has 6 missing values
- We can either drop the values or 'fill' the values with most appropiate values. Eye balling the 5 point summary might provide data for best analysis
5 Point Summary¶
In [10]:
# Eye ball the data
data_df.describe().transpose()
Out[10]:
Inference¶
- By analysing the above data, since dropping the rows might result in loss of data, its better to replace the missing values with median
In [11]:
# Make a copy of dataframe before replacing missing values with median; just in case the original dataframe requires to be revisited
data_df_Original = data_df.copy()
Replace the missing values with median¶
In [12]:
data_df.fillna(data_df.median(),axis=0,inplace=True)
In [13]:
data_df.info()
Eye ball the data again after replacing missing values¶
5 Point Summary¶
In [14]:
# Eye ball the data
data_df.describe().transpose()
Out[14]:
In [15]:
plt.figure(figsize= (30,20)) # Set the figure size
pos = 1 # a variable to manage the position of the subplot in the overall plot
totalrows, totalcols = data_df.shape
categorical_vars = data_df.describe(include=["object"]).columns
continuous_vars = data_df.describe(exclude=['object']).columns
for feature in data_df.columns:
plt.subplot(7, 3, pos) # plot grid
if feature in continuous_vars: # Plot histogram for all the continuous columns
sns.distplot(data_df[feature], kde= True,rug = True )
if feature in categorical_vars:
sns.countplot(data_df[feature], palette= 'Blues') # Plot bar chart for all the categorical columns
#sns.barplot(data_df[feature].value_counts().values, data_df[feature].value_counts().index)
pos += 1 # to plot over the grid one by one
In [16]:
# List numerical columns
numerical_cols = list(data_df.select_dtypes(exclude=['object']))
_= data_df.hist(column=numerical_cols, figsize = (16,16))
In [17]:
print(f"Skweness is: \n{data_df.skew()}")
Task 1 c. Outlier identification and appropriate actions¶
Plot box plots to visualize outliers easily¶
In [18]:
numerical_cols = list(data_df.select_dtypes(exclude=['object']))
_= data_df.boxplot(column=numerical_cols, figsize = (20,20))
In [19]:
plt.figure(figsize= (20,40))
continuous_vars = data_df.describe(exclude=['object']).columns
pos = 1
for feature in continuous_vars:
plt.subplot(10, 2, pos) # plot grid
sns.boxplot(data_df[feature])
# plt.title(feature)
pos += 1 # to plot over the grid one by one
In [20]:
# function to determine outliers of given series
def determineAndReplaceOutliers(aSeries, columnName):
q1 = aSeries.quantile(0.25)
q3 = aSeries.quantile(0.75)
iqr = q3-q1 #Interquartile range
threshold_low = q1-1.5*iqr
threshold_high = q3+1.5*iqr
outliers_low = aSeries[(aSeries < threshold_low)]
outliers_high= aSeries[(aSeries > threshold_high)]
#print ("Q1: ", q1)
print(f'threshold low: {threshold_low} \nlow outliers count =', outliers_low.count())
#print('Outliers low: \n')
#print(outliers_low)
data_df.loc[outliers_low.index, columnName] = threshold_low
print(f'replaced {outliers_low.count()} low outliers with (q1-1.5*iqr) = {threshold_low}')
#print ("Q3: ", q3)
print(f'threshold high: {threshold_high} \nhigh outliers count = {outliers_high.count()}')
#print('Outliers high: \n')
#print(outliers_high)
data_df.loc[outliers_high.index, columnName] = threshold_high
print(f'replaced {outliers_high.count()} high outliers with (q1+1.5*iqr) = {threshold_high}')
return threshold_low, outliers_low, threshold_high, outliers_high
In [21]:
columnName = 'radius_ratio'
threshold_low, outliers_low, threshold_high, outliers_high = determineAndReplaceOutliers(data_df[columnName],columnName)
In [22]:
columnName = 'pr.axis_aspect_ratio'
threshold_low, outliers_low, threshold_high, outliers_high = determineAndReplaceOutliers(data_df[columnName],columnName)
In [23]:
columnName = 'max.length_aspect_ratio'
threshold_low, outliers_low, threshold_high, outliers_high = determineAndReplaceOutliers(data_df[columnName],columnName)
In [24]:
columnName = 'max.length_aspect_ratio'
threshold_low, outliers_low, threshold_high, outliers_high = determineAndReplaceOutliers(data_df[columnName],columnName)
In [25]:
columnName = 'scaled_variance'
threshold_low, outliers_low, threshold_high, outliers_high = determineAndReplaceOutliers(data_df[columnName],columnName)
In [26]:
columnName = 'scaled_variance.1'
threshold_low, outliers_low, threshold_high, outliers_high = determineAndReplaceOutliers(data_df[columnName],columnName)
In [27]:
columnName = 'scaled_radius_of_gyration.1'
threshold_low, outliers_low, threshold_high, outliers_high = determineAndReplaceOutliers(data_df[columnName],columnName)
In [28]:
columnName = 'skewness_about'
threshold_low, outliers_low, threshold_high, outliers_high = determineAndReplaceOutliers(data_df[columnName],columnName)
Inference¶
- There are high outliers in radius_ratio
- Instead of dropping the rows; we have replaced the low outliers with (Q1 - 1.5 IQR) and high outliers with (Q3 + 1.5 IQR)
In [29]:
numerical_cols = list(data_df.select_dtypes(exclude=['object']))
_= data_df.boxplot(column=numerical_cols, figsize = (20,20))
From the above box plot, it is evident that the missing value and outliers of the data is take care of and there is no outliers now in the dataset¶
Task 2 : 2. Understanding the attributes - Find relationships between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why ¶
Task 2 a: 5 Point Summary¶
In [30]:
# Eye ball the data
data_df.describe().transpose()
Out[30]:
Task 2 b: Univariate Analysis & its distributions¶
In [31]:
plt.figure(figsize= (20,40))
columns_vars = data_df.drop('class',axis=1).columns #data_df.columns
pos = 1
for feature in columns_vars:
plt.subplot(10, 2, pos) # plot grid
sns.distplot( data_df[data_df['class'] == 'car'][feature], color = 'r')
sns.distplot( data_df[data_df['class'] == 'van'][feature], color = 'b')
sns.distplot( data_df[data_df['class'] == 'bus'][feature], color = 'g')
pos += 1 # to plot over the grid one by one
Inference¶
- Spread of compactness is least for van. mean compactness is highest for car. For Bus compactness is right skewed
- Mean circularity is higher for cars
- Mean distance_circularity is also higher for cars
- Mean radius_ratio is higher for cars, followed by Bus. It is least for vans
- pr.axis_aspect_ratio is has almost same distribution for car, van and buses
- max.length_aspect_ratio is almost same for cars and vans, lower for buses
- Mean scatter ratio is highest for cars, followed by bus and van
- Mean elomngatedness is highest for vans folowed by bus and car
- pr.axis_rectangularity is highest for cars, followed by bus and then vans
- distribution of max.length_rectangularity is almost same for cars, bus and vans
Task 2 c: Infer the insights about the relationship among the variables¶
In [32]:
sns.pairplot(data_df,hue='class',diag_kind='kde')
plt.show()
In [33]:
data_df.corr()
Out[33]:
In [34]:
plt.figure(figsize=(10,10))
sns.heatmap(data_df.corr(),
annot=True,
linewidths=.5,
center=1,
cbar=False,
cmap="YlGnBu")
plt.show()
Inference¶
- Mean scaled variance and Mean scaled variance1 is highest for cars followed by bus then vans
- 'scaled_radius_of_gyration', 'scaled_radius_of_gyration.1', 'skewness_about', 'skewness_about.1', 'skewness_about.2', have almost similar distribution for cars, buses and vans.
- 'hollows_ratio' is lower for buses as compared to cars and vans
- Many columns have long tails indicating outliers
- Scatter ratio & Scaled_variance1 has almost perfect positive linear relationship
- Many features show high correlation indicating that we might drop features which may be overlapping effect
- From above correlation matrix we can see that there are many features which are highly correlated.
- We can choose to drop one of highly correlated column but since we are attempting PCA, which is based on dimesionality reduction; we will NOT drop these columns
Task 2 d: Separate the Feature and Target variable¶
In [35]:
sns.countplot(data_df['class'])
plt.show()
In [36]:
data_df['class'].value_counts()
Out[36]:
Data transformations (Type changes/appropriate scaling of variables and encoding variables if required)¶
In [37]:
from sklearn.preprocessing import LabelEncoder
labelencoder = LabelEncoder()
data_df['class'] = labelencoder.fit_transform(data_df['class'])
data_df['class'].value_counts()
Out[37]:
In [38]:
# independant variables
X = data_df.drop('class',axis=1)
# the dependent variable
y = data_df['class']
print("shape of independent variable: ",X.shape)
print("shape of dependent variable: ",y.shape)
Task 2 e: Scaling of the data¶
In [39]:
# scale independent attributes
from scipy.stats import zscore
X = X.apply(zscore)
X.head
Out[39]:
In [40]:
covMatrix = np.cov(X,rowvar=False)
print(covMatrix)
Task 2 f: Train model using the train data with hyper parameters obtained¶
In [41]:
from sklearn.decomposition import PCA
pca = PCA(n_components=18)
pca.fit(X)
Out[41]:
In [42]:
# The eigen Values
print(pca.explained_variance_)
In [43]:
# The eigen Vectors
print(pca.components_)
In [44]:
# and the percentage of variation explained by each eigen Vector
print(pca.explained_variance_ratio_)
In [45]:
plt.bar(list(range(1,19)),pca.explained_variance_ratio_,alpha=0.5, align='center')
plt.ylabel('Variation explained')
plt.xlabel('eigen Value')
plt.show()
In [46]:
plt.step(list(range(1,19)),np.cumsum(pca.explained_variance_ratio_), where='mid')
plt.ylabel('Sum of variation explained')
plt.xlabel('eigen Value')
plt.show()
Inference¶
- Hyperparameters are obtained
- From above we can see that 8 dimension are able to explain 95% variance of data. so we will use first 8 principal components
In [47]:
pca8 = PCA(n_components=8)
pca8.fit(X)
print(pca8.components_)
print(pca8.explained_variance_ratio_)
Xpca8 = pca8.transform(X)
In [48]:
Xpca8
Out[48]:
In [49]:
# plot pair graph using PCA 8 parameters intead of all 19
sns.pairplot(pd.DataFrame(Xpca8))
Out[49]:
Task 3: Split the data into train and test (Suggestion: specify “random state” if you are using train_test_split from Sklearn)¶
Task 3 a: Split into Train and Test¶
Lets construct two SVM models. The first with all the 18 independent variables and the second with only the 8 new variables constructed using PCA.
In [50]:
# split data in 70 / 30 of all 18 dimesions
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, test_size=0.3, random_state=100)
X_train.shape,X_test.shape, y_train.shape, y_test.shape
Out[50]:
Task 4: Train a support vector machine using the train set and get the accuracy on the test set¶
In [52]:
from sklearn import metrics
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score,confusion_matrix
svm = SVC(gamma = 'auto')
model = svm
model.fit(X_train, y_train) # for the model with the train data
pred = model.predict(X_test) # make predictions on the test set
score_train = round(model.score(X_train, y_train), 3) # accuracy for train data
score = round(model.score(X_test, y_test), 3) # compute accuracy score for test set
print (score_train)
print (score)
print("Confusion matrix with raw data(18 dimension)\n",confusion_matrix(y_test,pred))
Task 4 b: Use Gridsearch to find appropriate hyperparametes¶
In [53]:
from sklearn.model_selection import GridSearchCV
from sklearn import svm
#making the instance
model=svm.SVC()
#Hyper Parameters Set
params = {'C': [0.01, 0.05, 0.5, 1],
# 'gamma':[0.01, 0.02 , 0.03 , 0.04, 0.05],
'kernel': ['linear','rbf']}
#Making models with hyper parameters sets
gs = GridSearchCV(model, param_grid=params, n_jobs=-1,cv=10)
gs.fit(X_train, y_train)
#The best hyper parameters set
print("Best Hyper Parameters:\n",gs.best_params_)
Task 5: Perform K-fold cross-validation and get the cross-validation score of the model¶
In [54]:
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
num_folds = 10
seed = 7
kfold = KFold(n_splits=num_folds)
model = gs
results = cross_val_score(gs,X_train,y_train, cv=kfold)
print(results)
cross_val_score_raw = "for raw data (18 dimension) cross_val_score Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0)
print (cross_val_score_raw)
In [55]:
prediction=gs.predict(X_test)
print("Accuracy:",metrics.accuracy_score(prediction,y_test))
accuracy_score_raw = metrics.accuracy_score(prediction,y_test)
#evaluation(Confusion Metrix)
print("Confusion Matrix:\n",metrics.confusion_matrix(y_test,prediction))
Test Accuracy (95.66%) is well in range of 95% confidence interval (86.8% to 99.6%)¶
Task 7: Repeat steps 3,4 and 5 but this time, use Principal Components instead of the original data. And the accuracy score should be on the same rows of test data that were used earlier. (hint: set the same random state)¶
In [56]:
# for PCA 8 dimensions
# split data in 70 / 30 of 8 dimesions
from sklearn.model_selection import train_test_split
X_train_pca8, X_test_pca8, y_train_pca8, y_test_pca8 = train_test_split(Xpca8, y, train_size=0.7, test_size=0.3, random_state=100)
X_train_pca8.shape,X_test_pca8.shape, y_train_pca8.shape, y_test_pca8.shape
Out[56]:
In [57]:
svm = SVC(gamma = 'auto')
model = svm
model.fit(X_train_pca8, y_train_pca8) # for the model with the train data
pred_pca8 = model.predict(X_test_pca8) # make predictions on the test set
score_train_pca8 = round(model.score(X_train_pca8, y_train_pca8), 3) # accuracy for train data
score_pca8= round(model.score(X_test_pca8, y_test_pca8), 3) # compute accuracy score for test set
print (score_train_pca8)
print (score_pca8)
print("Confusion matrix with PCA data (8 dimension)\n",confusion_matrix(y_test_pca8,pred_pca8))
In [58]:
from sklearn.model_selection import GridSearchCV
from sklearn import svm
#making the instance
model=svm.SVC()
#Hyper Parameters Set
params = {'C': [0.01, 0.05, 0.5, 1],
# 'gamma':[0.01, 0.02 , 0.03 , 0.04, 0.05],
'kernel': ['linear','rbf']}
#Making models with hyper parameters sets
gs = GridSearchCV(model, param_grid=params, n_jobs=-1,cv=10)
gs.fit(X_train_pca8, y_train_pca8)
#The best hyper parameters set
print("Best Hyper Parameters:\n",gs.best_params_)
In [59]:
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
num_folds = 10
seed = 7
kfold = KFold(n_splits=num_folds)
model = gs
results = cross_val_score(gs,X_train_pca8,y_train_pca8, cv=kfold)
print(results)
cross_val_score_pca8 = "for pca (8 dimension) cross_val_score Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0)
print(cross_val_score_pca8)
In [60]:
prediction_pca8=gs.predict(X_test_pca8)
print("Accuracy:",metrics.accuracy_score(prediction_pca8,y_test_pca8))
accuracy_score_pca8 = metrics.accuracy_score(prediction_pca8,y_test_pca8)
#evaluation(Confusion Metrix)
print("Confusion Matrix:\n",metrics.confusion_matrix(y_test_pca8,prediction_pca8))
Test Accuracy (92.12%) is well in range of 95% confidence interval (86.8% to 99.6%)¶
Task 8. Compare the accuracy scores and cross-validation scores of Support vector machines – one trained using raw data and the other using Principal Components, and mention your findings (10 points) ¶
In [61]:
print(f'for raw data (18 dimension) accuracy_score: {accuracy_score_raw}\n{cross_val_score_raw}')
print(f'for pca (8 dimension) accuracy_score: {accuracy_score_pca8}\n{cross_val_score_pca8}')
Inference¶
- Looking at the confusion matrix, model predicts all the vans correctly through Silhoutte
- From above we can see that PCA is performing well. The dimesions was reduced from 18 dimesions to 8 dimension.
- Accuracy with pca is about 92% and with raw data about 95%
- Please note that pca 92% accuracy is with only 8 dimensions where as rawdata has 18 dimensions
- Generally we determine whether a given model is optimal by looking at it’s F1, precision, recall, and accuracy (for classification), or it’s coefficient of determination (R2) and error (for regression)
- In K Fold cross validation, the data is divided into k subsets. Now the holdout method is repeated k times, such that each time, one of the k subsets is used as the test set/ validation set and the other k-1 subsets are put together to form a training set. This significantly reduces bias as we are using most of the data for fitting, and also significantly reduces variance as most of the data is also being used in validation set.
- The cross validation score for raw (18 dimensions) is 96.78% (with SD 1.92%) and PCA for 8 dimension is 95.43% (with SD 1.52%)
In [ ]: